Mixed reality, which can also be known as augmented reality, involves the merging of real-world objects and/or people with virtual objects to produce new environments and visualizations where physical and digital objects co-exist and interact in real time. Mixed-reality devices augment a user's view of the real world with virtual objects that aim to look as if they are actually placed within the real world. A mixed-reality device may allow the user to view their real-world surroundings through a semitransparent display. Virtual objects can then be presented on the display. These virtual objects appear to be superimposed on the user's view of their real-world surroundings, thereby merging virtual reality with physical reality.
A mixed-reality experience can be shared among multiple mixed-reality devices. This enables multiple users to have common shared experiences within a shared mixed-reality environment. There are many different scenarios where a shared mixed-reality experience could be useful and/or enjoyable, such as a game where players are able to interact with one another and with virtual objects as part of the game.
To facilitate shared mixed-reality experiences, it is important for multiple devices to be able to compute their position and motion in the same coordinate system so that they can be aware of each other's relative position. Most mixed-reality devices are able to determine their own relative motion, but not necessarily their relative pose (i.e., position and orientation) with respect to other devices.
The typical solution to register the coordinate system for multiple devices is to exchange three-dimensional map information (or two-dimensional image data) between devices so that the relative pose between these maps/images can be determined. These solutions can either be implemented peer-to-peer or over a cloud service.
The exchange of map data, however, can be cumbersome and present privacy risks. Moreover, when users' viewpoints differ significantly from each other the traditional image feature matching can fail, resulting in the inability of devices to determine a joint coordinate system and thus to share a mixed-reality experience. This may happen, for example, in scenarios where users look at each other and the camera and therefore view opposite sides of the same space, which is often the natural configuration for games and other types of shared mixed-reality experiences.